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ABSTRACT 

This study reports the results of a survey sent to 67 
metropolitan school district evaluators. The survey assessed past and 
anticipated conduct of cost analysis methods, as well as attitudes 
toward the use of these methods. The instrument used contained many 
items taken from a survey instrument used in a previous study of cost 
analysis methods at state education agencies. Results indicated that 
the number of school district evaluation units required to conduct 
some type of cost analysis study is expected to increase over 50 
percent in the next five years, as well as an anticipated increase in 
all five types of cost studies (cost description, cost feasibility, 
cost utility analysis, cost benefit analysis, and cost effectiveness 
analysis). The major impediments to using these methods were: (1) 
results are not complete; (2) they do not help improve program 
operations; (3) outcome data are usually not available; and (4) 
institutional requests for cost studies are insufficient. A separate 
attitudinal scale for cost-effectiveness methodology also revealed 
methodological impediments. However, the data suggest that as the 
number of cost analysis studies increases/ the evaluators' attitudes 
toward these methods may become more positive. Appendices contain 
samples of survey letters and the survey instrument itself. (DWH) 
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PREFACE 



The Research on Evaluation Program is a Northwest Regional 
Educational Laboratory project of research, development, testing, 
and training designed to create new evaluation methodologies for 
use in education. This document is one of & series of papers and 
reports produced by program staff, visiting scholars, adjunct 
scholars, and project collaborator s*--*all members of a cooperative 
network of colleagues working on the development of new 
methodologies. 

To what extent are local school districts using cost analysis 
methods in their evaluation work? What problems are the 
districts having in using cost methods? and what are the 
prospects for future use? These and related questions are 
answered in this report of cost analysis activities conducted by 
a national s£uaple of 67 metropolitan school districts. 

Nick L. Smith., Editor 
Paper and Report Series 
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COST ANALYSIS AT THE LOCAL LEVEL: 
APPLICATIONS AND ATTITUDES 



Introcluctlon 



There are currently many indications that cost analysis is 
being recognized as a useful method in educational evaluation. A 
recent survey o£ educational evaluators showed that over the next 
five years nearly 60 percent of state level evaluation units will 

required to conduct cost analysis studies (Smith, N. L. & 
Smith, J. K. , 1984). The number of cost analysis questions asked 
of consultants by local districts and state departments of 
education is increasing (Gray, P. J. & Smith, J. K. , 1983). 
Evaluators are requesting information on cost analysis methods in 
the form of special training sessions at conferences (B# Ingle, 
personal communication, October 10, 1984), and written materials 
(Smith, N» L. & Smith, J. K. , 1984). The number of published 
articles on cost*-analysis applications is increasing yearly 
{e.g., see Smith, J. K. & Smith, N. L. , 1983 for a bibliography 
of over 350 references to the use of cost analysis in evaluation, 
and Warner, K. & Hutton, R. , 1980, for a review of cost analysis 
articles in health evaluations) • 

The study reported here is one in a series of four studies 
designed to determine the state of practice of cost analysis 
methods in educational evaluation, and to identify attitudinal 
factors which might be affecting application of the methods. The 
first in the series of studies looked at the conduct of cost 
analysis studies by e^n educational research and develoE<fater(t 
laboratory (Smith, J. K« , 1983). The second study included an 
analysis of published examples of cost analysis evaluations 
(Smith, N. L. & Smith, J. K. , 1984). The third study explored 
applications of cost analysis in state department of education 
evaluation units (Smith, N. L. 6 Smith, J. K. , 1984). This 



fourth study focuses on applications of cost analysis methods by 
evaluation units in school districts. 

This report describes the results of a survey sent to 67 
metropolitan school district evaluators. The survey assessed 
past and anticipated conduct of cost analysis methods^ as well as 
attitudes toward the use of these methods. The report is divided 
into four major sections: (1) Procedure^ (2) Results: 
Applications^ (3) Results: Attitudes^ and (4) Conclusion. 

Procedure 

Sampling Method 

Over 15^000 school districts (also referred to as local 
education agencies (LEAS)) in the United States enroll from 1 to 
900^000 students. The intent of this study was to look at cost 
studies as they are conducted by evaluation units in local school 
districts r rather than to look at which districts do and do not 
conduct cost studies. Lyon^ Doscher^ McGranahan^ and Williams 
(1978) 9 found that the existence of a formal evaluation unit in 
an LEA is positively related to the size of the district. 
Specif icallyr 89 percent of LEAs in metropolitan districts 
(45^000 or more students) have evaluation units^ while only 59 
percent of LEAs in large districts (25r000-44|999 students) and 
33 percent of LEAs in raedium-oized districts (10|000-24|999 
students) have evaluation units. Because metropolitan districts 
are most likely to have formal evaluation units, we chose to 
survey this group for our study. 

The names and addresses of directors of LEA evaluation units 
which serve districts of 45|000 or more students were obtained 
from the Lyon et. al. study (1978). This list was updated to^ 
reflect the names of current evaluation unit directors and - 
address changes. In all, 67 metropolitan districts with formal 
education units were identified. Surveys were sent to these 67 
units. 



The survey method used was the personalized approach combined 
with repeated mailings, as advocated by Oillman (1978). The 
survey (see Appendix), along with a personalized cover letter 
(see Appendix), and a self*-addressed, stamped, return envelope, 
were mailed to each evaluation unit director. One week later a 
postcard reminder was sent (see Appendix). After four weeks, 29 
completed surveys had been received. 

At this time, another personalized letter (see Appendix), the 
survey, and a return envelope were mailed to the 38 directors who 
had not returned the survey. Five completed surveys were 
received as a result of this effort. Two weeks later a telephone 
call was made to each of the 33 remaining nonrespondents to 
determine whether 'the survey had ever been received, and whether 
it would be returned* Eighteen completed surveys were returned 
following the telephone calls. 

In total, 29 surveys were returned after the initial letter 
and postcard, and 23 were returned after the second letter and 
telephone call, resulting in 52 completed surveys. Each 
evaluation unit returning the survey was ser a thank'-you letter 
containing a summary of the study results (see Appendix), and a 
complimentary copy of Henry Levin's new book. Cost Analysis; 
A Primer (1983). 

Instrument Development 

A survey booklet containing four major sections was used in 
this study. Most survey items were taken from a survey 
instrument used in an earlier study of cost analysis methods at 
state education agencies (Smith, N. L. & Smith, J. K. , 1984). 

The first section of the survey was designed to assess the 
conduct of past cost analysis studies, and to obtain projections 
of anticipated cost studies. The second section contained a 
scale that measured attitudes toward the conduct of cost analysis 
methods in general. This scale was shown to be reliable in the 
state education agency (SEA) study by a standardized alpha 
coefficient of .64. The third section of the survey contained a 
scale that measured attitudes toward the conduct of cost 
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effectiveness analysis in particular. Cost-effectiveness 
analysis has been suggested to be the roost appropriate cost 
analysis method for educational evaluation {Levin, 1983), and we 
were interested in determining evalua tors' attitudes toward this 
method. The cost-effectiveness attitude scale, also developed 
for the SEA study, had a reliability of .73. The final section 
of the survey contained questions about characteristics of the 
evaluation unit and about the responsibilities assumed by its 
staff. 

Sample Description 

Of the 67 metropolitan school districts with evaluation units 
in 1983, 52 (78%) returned completed surveys. A check was made 
to identify nonrespondent bias resulting from the 15 surveys not 
returned. According to the National Center for Education 
Statistics for 1981-1982, the average enrollment of these 15 
districts was 74,403 students. In contrast, according to the 
same source, the average enrollment of the districts which 
returned the survey was 97,792. Because of these size 
differences, the results of this study may be slightly biased in 
favor of larger districts in the metropolitan size category. 

The average metropolitan school district evaluation unit had 
9.6 full-time staff and a budget of $596,101. In order to 
determine the range of responsibilities assumed by these units, 
respondents were asked to check, from a list of 10, all major 
cesponsiblities assumed by staff in their unit. The primary 
responsibilities of these staff were: 



conduct of evaluation studies 


92.3% 


n 


SI 


48 


consultation and technical as.sistance 


88.8% 


n 


adc 


48 


research studies ^ 


80.8% 


n 


3 


42 


the conduct of needs assessments 


78.8% 


n 


3 


41 


running testing programs 


76.9% 


n 


ac 


40 


evaluation monitoring 


75.0% 


n 




39 


provision of information 


71.2% 


n 


3 


37 



Less than half of the LEAs were involved in planning activities 
(48.1%, n a 25), or policy analysis (42,3%, n « 22). 



Results; Applications 

Requirements to Conduct Cost Studies 

To obtain a picture of current requirements to conduct cost 
analysis studies, we asked whether there was a formal expectation 
or requirement within. ^e district to conduct 89me form of cost 
analysis work. Only 21 percent of the districts (n » 11) were 
currently required to conduct cost analysis studies, and an 
average 3.3 percent of their budget was allocated to this type of 
analysis. We then asked whether a formal expectation or 
requirement to do some form of cost analysis was anticipated 
within the next 5 years* The percent of LSAs expecting to be 
required to conduct cost analyses over the next 5 years increased 
from 21 percent to 71 percent (n ";37). Further, they 
anticipated that an average of 9 percent of their total budget 
would be allocated to the conduct of cost studies in the next 5 
years. 

When asked to explain the increase of requests for this type 
of analysis, some respondents said, "Limits on resources and the 
climate of high need for accountaliility lead us to expect mote 
formal requests for cost analyses;" "Cost analysis is becoming 
more essential due to shrinking funds and demands of the public 
that all programs be analyzed in terms of cost**effectiveness;" 
and "[Cost analysis] work weWe done to date has been well 
received. " 

Those units that had not been required to conduct any cost 
studies in the past five years were asked to indicate the primary 
reason no cost studies had been conducted. . In these open**ended 
questions, two reasons stood out: (1) 10 respondents said that 
they were not asked to conduct studies, or that their decision 
makers were not interested in cost analysis data; (2) five 
respondents said that other units conducted cost studies (e.g., 
budget or accounting departments). Additional comments included, 
''Decision makers don't have a gcod concept of what cost studies 
produce for their use. They don't understand what they are, or 
how to use them;" and "Studies have been done, but these cost 



studies were basically done by finance and operations people. 
Other reasons for not conducting cost studies ' included time and 
staff limitations (n » 3) ; the lack of knowledge about coat 
analysis methods (n « 2) ; projec^t mandates which would negate 
cost analysis results (n » 2) ; cbanges in office staff (n » 1)^ 
ana unavailability of data^(n ■ 1) . > 

i\pplicationa of Cost Analysis Methods 

We wanted to determine the purposes for conduct 4ng cost 
analysis studies # and alsked respondents to indicate the number of 
five different cost analysis methods they had conducted in the 
past^ or expec':ed to conduct in the future. Table 1 describes 
the five cost analysis methods, and' gives the flumber of these 
methods which have been conducted and the number anticipated. 

Clearly, the most frequent reason to conduct a cost amalysis 
is to ^Jesc^ibe the costs ^of a program or programs* This method 
has beeh used, and is expected to continue to be used, .with 
significantly more frequency than any other method. The average 
number of cost descriptive studies is projected to increase from 
8.9 to 16.4 over the next five years. Of special interest here 
is the large range in the number of past and anticipated cost 
descriptive studies, as compared to the range in the number of 
other types of cost studies. One explanation for this large 
number of cost descriptive studies is that some respondents may 
have perceived the maintenance of program budgets as a cost 
description. In that case, the number of cost descriptions may 
equal the number of programs sponsored or supervised by tjhe 
evaluation unit. 

The number of times each of the other types of cost analysis 
methods have been conducted is much smaller, although an increase 
is expected for each over the next five years. The method 
conducted with the second highest frequency is cost feasibility. 
Of the three cost-outcome me thodsr^ cost-effectiveness analysis is 
projected to be conducted most often, followed by cost-utility 
and cost-benefit analysis. 
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Table 1 

Mean Number of Past and Anticipated 
Cost Studies Conducted by LEA Evaluation Units 



^ 



Mean Number o£ Cost Studies 



Pufpose o|E Cost Study 

a. To describe the costs of a 
program or programs 

(cost description) 

b. To compare the cosis qf a 
program or programs with 
resources available to see 
if they are affordable 
(cost feasibility) 

c. To Compaq costs and 
outcomes for 2 or more 
programs, where outcomes 
are estimated 

(cost utility analysis) 

d. .» To compare the costs and 

outcomes for 2 or loore 
programs, where outcomes 
are measured in dollars 
(cost benefit analysis) 

e. To compare the costs and 
outcomes for 2 or more 
programs, where outcomes 
are measured in test 
scores , behavioral 
racings, etc. 

(cost effectiveness analysis 



Done in past 
5 years 
(range) 



8.9 

(0-225) 



2.2 

(0-50) 



.54 

(0-5) 



.46 

(0-10) 



Anticipated 
over the 
next 5 years 
(range) 

16.4 

(0-205) 



3.3 

(0-50) 



1.4 

(0rl2) 



.92 
(0-24) 




2.7 
(0-40) 



At first glance, the number of cost utility, 
cost-effectiveness, 4md cost benefit studies appear to be very 
few in comparison to the number of cost descriptive and cost 
feasibility studies. For example, according to these figures, 
each LEA will be conducting an average of two-to-three 
cost-effectiveness analyses, as compared to about 16 cost 
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descriptive studies over the next 5 years. In terms of the 
implementation of theS?- methods, however, this number of 
cost-effectiveness studies is quite realistic. Levin (1983) has 
said that even the simplest c'ost^-ef fectiveness study takes at 
least six months to conduct. Our ext^erience shows \that even six 
months may be an optimistic estimate. In light of the time 
required to conduct a cost-effectiveness study, two-three per 
district over a five-year span is an ambitious task. 

Descriptions of Cost Studies Conducted 

We provided space on the survey for respondents to describe 
^what they considered the "best" example of a cost analysis 
conducted in their unit during the past five years. Thirty 
"best" studies were described. In terms of the content area of 
these studies, eight studies were conducted on reading programs. 
The other descriptions included studies of program components 
(e.g., the cost of aides^ versus teachers, n » 6) , analysis of 
special programs (e.g., summer school, n * 4) , alternative time 
schedules (n » 3) , alternative math programs (n « 3) , facility 
costs (n « 2) , and one analysis each of transportation 
alternatives, testing alternatives, special education 
alternatives, and science program alternatives. 

These "best" examples most frequently used cost-effectiveness 
analysis (n » 10) which has been identified as the roost 
appropriate method for educational evaluation (Levin, 1983). 
Other methods used included cost description (n » 9) , cost 
benefit (n » 5) , cost feasibility (n » 4) , and cost utility 
(n - 2). 

We also asked for an indication of the value of the study, 
and how the results were used for decision making. The vast 
majority said that the study results were put into action (n = 
22), and that the study was very valuable in decision making (n » 
19). Only four respondents said that the study had not been of 
value in decision making. 
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Summary 

According to these respondents # the number o£ school dX^^trict 
evaluation units required to conduct some type of cost analysis 
study will increase over the next five years by 50 perceiw. 
Cor respond inglyT' the data show an anticipated increase in^ the 
number of all five types of cost studies, especially cost 
descriptive studies. The next section focuses on attitudes about 
cost analysis methods and how these attitudes might affect the 
conduct of cost analysis studies. A discussion of attitudes 
about cost analysis methods in general is followed by a 
discussion of attitudes about cost-e|Efectiveness analysis in 
paf ticular • 



Results; Attitudes 

Attitudes Toward Cost Analysis 

In order to assess attitudes toward the use of cost-analysis 
in educational evaluation, respondents were asked to complete a 
general scale of "impediments" toward the use of these methods. 
This scale had been developed during a previous study of 
applications of cost*-analysis at the state level. The scale 
items were derived through our earlier research, and through 
identification of impediments cited in the literature. The 
empirical basis for identification ^of the impediments helped to 
insure proper face and content validity for the scale. Further, 
following analysis of the state level evaluators responses, a 
reliability coefficient (standardized item Alpha) was computed at 
.64. Because of the demonstrated reliability and apparent 
validity, this scale was used in the current study. 

The reliability of this scale for the local level evaluators 
was again high, as shown by the standardized alpha coefficient of 
.76. Further, respondents agreed that many scale items were 
indeed an impediment r as shown by the ratings in Table 2. The 
greatest impediments to the conduct of a cost analysis study were: 
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Mean Don't Know 
We are seldom asked to do cost studies 3.26 0 

Cost study results are incomplete because 3.17 4 

it is not possible to include all 
important cost factors 

Cost results do not tell managers how 3.07 5 

to improve program operations 

Accurate outcome data are usually not 3.05 4 

available 

Three of the four greatest impediments have to do with 
methodological problems with cost analysis methods: the results 
are not complete; they do not help improve program operations; 
and outcome data are usually not available. The largest 
impediment, however, is of a more political nature: until therf.- 
is administrative and institutional support for cost analysis 
studies, and until more of them are requested, these studies are 
unlikely to be conducted. 

Another methodological problem, the lack of availability of 
cost data, correlated highly with several other impediments on 
the scale. For example, the most highly correlated it'ims were: 
studies take too much time and cost data are usually not 
.available (d with f » .62); studies cost too much to conduct and 
cost data are usually not available (e with f .65); and results 
are incomplete because it is not possible to include all cost 
factors and cost data are usually not available (g with f « .61). 

Attitudes toward Cost Effectiveness Analysis 

The second attitudinal scale dealt specifically with 
cost-effectiveness analysis. This scale was developed in the 
same manner as that of the general cost analysis scale, using 
previous research experience and the literature to derive the 
impediments items. This maximized the content and face / 
validities of the scale. In addition, analysis of the state 
level data for this scale resulted in a standardized alpha 
coefficient of .73. 

10 lb' 



Table 2 



Impediments to the Use of Cost analysis in 
LEA Evaluation Units 



Item Mean* 

a. We are seldom asked to do cost studies 3.26 

b. Decision makers are not often interested in 2.76 
actual cost information 

c. Decision makers do not often use the results 2.44 
of cost studies 

d. It takes too much time to conduct cost studies 2.84 

e. It costs too much to conduct cost studies 2«48 

f. Accurate cost data are usually not available 2.82 

Cost study results are incomplete because it 3.17 
is not possible to include all important 
cost factors 

Accurate outcome data are usually not 3.05 
available 

i. It is difficult to relate cost data to 2.78 
educational outcomes 

m 

j. Cost results do not tell managers how to 3«07 
improve program operations 

k. We lack staff with the technical capability 2.51 
to conduct cost studies 

1. We lack available consultants or experts to 2.36 
help us conduct cost studies ^ 

m« We do not have sufficient experience in 2.63 
conducting cost studies 

n. We have few guidebooksr texts^ or examples 2.90 
to follow in conducting cost studies 

* 4 « Strongly Agree 
3 * Agree 
2 » Disagree 
1 « Strongly Disagree 

Mean is based on N of 52 minus number of Don't Knows 
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An analysis of the responses of district level evaluators 
showed a standardized alpha coefficient of .86. Their ratings of 
the impediments on this scale are shown in Table 3. Five out of 
eight items weta perceived as impediments by the respondents, as 
indicated by mean ratings of ?.0 or above. Like the impediments 
to the conduct of cost analysis studies in general » the 
impediments to the conduct of cost-effectiveness analyses were 
methodological in nature. These impediments were that the method 
was costly and complex (m » 3,30), lii^lted in assessing outcomes 
(m » 3.53), provided no information on local conditions 
(ra * 3.40), too technical (m ■ 3.34), and needed methodological 
developftent (m » 3.30). Interestingly, the decision makers' 
disinterest in cost-analysis was the least of all the impediments. 

Again, the items roost highly intercorrelated all had to do 
with the methodology involved in the conduct of a cost- 
effect iveV^ess analysis. That cost-effectiveness was seen as 
costly and complex was related to its limitations in assessing 
multiple program outcomes (d with e - .63); the complexity of 
cost-effectiveness analysis was also related to the technical 
difficulty of the method (d with g » .63); finally, the 
complexity was related to the need for further development of the 
method (d with h « .67). 

The perception that cost-effectiveness methodology ne«ds 
development was related to the perception that cost-effectiveness 
gives no information on program procedures or local conditions 
(h with f » .60); the need for method development was also 
associated with the perception that the method is too technical 
(h with g « .62). 



Conclusion 
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The data show that cost-analysis methods are anticipated to 
be a part of local evaluations in the future. Although only 21 
percent of local evaluation units have had a formal requirement 
to conduct cost studies in the past, this is expected to increase 
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Table 3 



Impediments to the Use of Cost Effective 
Analysis in LEA Evaluation Units 



Item I Mean* 

Cost-^ef fectiveness analysi<^. is often unneces- 2.34 
sary because decision makers are not 
interested in relatin9 prcgraun costs and 
effects 

Cost-effectiveness analysis is often not 2.86 
feasible because comparative program studies 
are seldom possible 

Cos t'-ef feet iveness analysis is too specialised 2.71 
a technique to be generally applicable 

Cost-effectiveness analysis is so costly and 3.30 
complex that it is warranted only for major 
studies 

Cost-effectiveness analysis considers only a 3.53 
limited number of program outcomes and so 
does not represent true program effects 

Cost-effectiveness analysis is of limited 3.40 
utility since it provides no information on 
program procedures or local conditions 

Cost-effectiveness analysis is difficult to 3 .34 

do because of technical details (e.g., 
discount rates) and the need for sophisti- 
cated analysis procedures 

Cost-effectiveness analysis cannot be applied 3«30 
to educational programs without further 
development of the method 



4 » Strongly Agree 
3 ■ Agree 
2 > Disagree 
1 " Strongly Disagree 

Mean is based on h of 52 minus number of Don't Knows 
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to 71 percent over the next five years. To quote one respondent, 
"As managers become more familiar with cost-analysis work, more 
requests will be made." 

In a final analysis, we attempted to identify relationships 
between evaluation unit characteristics, attitudes, and past and 
anticipated conduct of cost analysis studies. Correlation 
matrices among unit descriptors (e.g., budget, number of staff, 
district size, responsibilities), attitudinal scale items, and 
experience with the methods (e.g. , total number of cost studies 
conducted; total number of effectiveness studies) were conducted. 

No unit descriptors correlated significantly with either past 
or anticipated studies. However, experience with the methods in 
the past was significantly related to the anticipated conduct of 
the methods in the future. There was a significant correlation 
between the number of past cost-effectiveness studies and the 
number of anticipated cost-effectiveness studies (r « .83, 
p » .001). Similarly, the total number of all cost studies 
correlated significantly with the total number of anticipated 
studies (r » .84, p - .001). Those who had conducted cost- 
analysis studies in the past were roost likely to conduct such 
studies in the future. 

Experience with the methods was also related to the 
perception of impediments or problems with implementing both 
cost-analysis studies in general and cost-effectiveness analyses 
in particular. Every impediment on the general cost analysis 
index correlated negatively with the total number of past cost 
studies conducted. That is, as the number of studies conducted 
increased, the perception of impediments decreased. Certainly, 
not all of the correlations were significant, but all were 
negative, and the trend was clear. 

Negative correlations were also observed between the number 
of past cost-effectiveness analyses conducted and perception of 
impediments in conducting a cost-effectiveness analysis. Again, 
all the correlations were negative, suggesting that, as the 
number of cost-effectiveness studies conducted increases, the 
perception of impediments decreases. 
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Another interesting observation on attitudes towarc 
cost~ef£ectiveness analysis was also related to respondents' 
experience with the method. The largest impediments to the 
conduct of cost-effectiveness analysis also had the largest 
number of ''don't know" responses. For example, while only one 
respondent didn't know whether decision makers were interested in 
relating program costs and effects, 10 or more didn't know 
whether the methodological issues described in items d - h were 
an impediment. This number of "don't know" responses was not 
found on the general cost analysis scale. It appears that some 
respondents did not have enough experience with cost- 
effectiveness analysis to be able to rate methodological 
impediments to the analysis. 

In summary, evaluators expect to be conducting more cost 
analysis studies in the upcoming five years. This increase may 
affect the way evaluators view cost analysis methods. The data 
suggest that, as the number of cost analysis studies increases, 
evaluators^ attitudes toward the methods may become less 
negative* Given that large increases are anticipated in the 
number of requests for cost analysis studies, more evaluators may 
endorse cost analysis methods as reasonable and useful tools for 
educational evaluation over the next few years. 
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rKXUnMMI 

Regional 

Educational 

Laboratory 




Ijetter accompanying survey 



300 S.W. Sixth Avenue • Portland* Oregon 97204 
TELEX: 701716 CABLE: NWREL SOURCE: STL0B8 



(503) 248-6800 



May 18, 1984 



Dear 

Are evaluators in your evaluation' unit being asked questions like "Can we 
really afford this new program?" or "Which of the two programs helps 
students most for the least money?" School staff and administrators 
dealing with declining school resources are increasingly asking local 
district evaluators such questions. 

We at the Northwest Regional Educational Laboratory are working to help 
school district evaluation units look at educational costs and include 
them where appropriate in their evaluation work. We are doing this by 
studying how evaluators currently use costs, by developing more efficient 
and practical cost methods, and by providing support materials for 
district evaluator use. 

In order to aid evaluators like yourself deal with costs, we need your 
help. We would appreciate it if you would take a few minutes to complete 
the enclosed survey. We believe that by answering as a representative 
for your evaluation unit, you can provide us with important information 
about the use of coat analysis methods in educational evaluation. 

We know that some evaluation units have had little or no experience in 
doing cost studies. Even if this is the case for your unit, your answers 
are important to us. Your evaluation unit was chosen because of your 
experience and the size of your district — we are interested in why 
metropolitan evaluation units like yours do or do not use cost data. 
Therefore, please complete this survey even if your experience with cost 
studies is limited. 

We will keep your answers completely anonymous. Each survey contains an 
identification number for mailing purposes only. This is so that we can 
check your name off the mailing list when your survey is returned. If 
you would like to receive a summary of the results of this study, write 
"copy of results requested" on the back of the return envelope, and print 
your name and address below it. Please do not put this information on 
the survey itself. . 
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AN EQUAL OPPORTUNITY EMPLOYER 



May IS, 1984 
Page 2 



Please complete the survey today^ if possible^ and return it in the 
enclosed self-addressed stamped envelope. We need to begin tabulating 
responses soon. If you have any questions or conunents on this survey^ 
please do not hesitate to write or call 1-800-54 7-6339. 

Thank you for your assistance. 

Sincerely I 



Jana Kay Smith, Ph.D. 
Research on Evaluation Program 

Enclosures 




Followup Postcard 



Harch 22 r 1984 

Last tfttk M stnt you ■ qu«iiticmnairt on tht um of cost analysis Mthods 
in your tvaluation unit. 

If you havt alrtady stnt it baek^ pltast acctpt our slncart thanks. Your 
contribution to this study is isost apprsciattd. If you havt not ytt 
rtturntd itf pltast do so today. Btcaust tbt qutstionnairt was stnt to 
only ont ptrson in your distriet* it is tsstntial that wt havt your 
qutstionnairt if wt art to undtrstand hon Mtropolitan school districts 
do and do not ust eott data in tvaluations. 

If by 9om chanct you did not rtctivt tht qutstionnairt # or it got 
Aisplsctd^ pltast call us right now* on our toll'-Crtt nuabtr 
l-800-547-M39f and wt will gtt anothtr ont in tht sail iiMtdiattly. 

Thank you (or your cooptration. ip > y ^ 




Rcttarch on Evaluation Program 



Rtstarch on Evaluation Program 
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NorthwMt 
Regional 
Educational 
Laboratory 




Followup Letter 



300 S.W. Sixth Avenue ♦ Portland, Oregon 97204 • (503 ) 248-^800 
TELEX: 701716 CABLE: NWREL SOURCE: STL058 



April 4, 1984 



Dear 

About three weeks ago you received a survey on the use of cost analysis 
methods in educatio?ial evaluation* As of yet we have not received your 
completed survey. 

The study looks at the practice of cost analysis in school district 
evaluations and at factors that might affect the conduct of cost analyses^ 

We are/wxiting to you again because of the importance of including y6ur 
survey! in our study. (Jnly one survey was sent to your school district. 
Consequently^ it is essential that you return your survey if the results 
are to accurately portray the use of cost analysis methods in school ^ 
districts. 

In the event that your survey has been misplaced^ we have enclosed 
another copy. If you have any questions or comments ^ please dpn^t 
hesitate to write or call 1-800-549-6339. 

Your cooperation is greatly appreciated. 



Sincerely^ 

jtna Kay Smithy Ph.D. 
Research on Evaluation Program 



Nick L. Smiths Ph.D.^ Director 
Research on Evaluation Program 



Enclosures 
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Thank-you Letter to Respo^fidei^ts 



November 19 ^ 1984 



Dear : 

Several months ago you assisted us with a survey about the use of cost 
analysis methods in metropolitan school dlstiict evaluation units. We 
are very grateful for the help you provided us, and are writing to share 
with you some of the results of the stud]^. 

Of the 67 metropolitan school districts we surveyed, 52 (78%) completed 
our questionnaire. Most evaluation units had had some experience with 
cost analysis methods, particularly with cost descriptions and cost 
feasibility analyses. Although, in general, the number of studies done 
was low and the methods used were relatively simple, the units doing cost 
studies reported being pleased with the impact qf their efforts. 

At the time of the survey, 21 percent of the units said there was 
currently a formal expectation or requirement that they do some form of 
cost analysis work, and they estimated devoting an average of 3.3 percent 
of their budgets to cost work. When asked to project future requirements 
to conduct cost analysis work, 71 percent anticipated having a formal 
requirement to do cost studies, and expected to spend an average of 
9 percent of their budgets for that purpose. Clearly, the respondents 
anticipate conducting more cost studies in the future. 

We also looked at the impediments or problems in conducting cost analysis 
studies. We found that as experience increased (measured as total number 
of cost studies conducted in the past 5 years) , the impediments or 
problems in implementing cost studies decreased. **Since the number of 
cost studies conducted by school districts is expected to increase over 
the next few years, we would expect evaluators to have fewer problems in 
doing cost studies. 

We are continuing our work on cost analysis methods in 1985, including 
developing more stref <t!iiir.ed methods for evaluation use, and producing 
guidebooks and examples ^o help evaluators conduct cost analysis 
studies. We welcome your suggestions, questions, and requests for 
materials* 
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November 19, 1984 
Page 2 



Enclosed is a copy of Henry Levin's new book Cost-Effectiveness;, 
A Primer which was developed in part with our support. We would also be 
happy to send you a copy of the full survey report, "Cost Analysis at the 
Local Level: Applications and Attitudes," upon request. Thank you again 
for participating in our survey study. 

Cordially, 



Jana Kay Smith, Ph.D. Nick L. Smith, Ph.D. 

Research on Evaluation Program 

JKS/NLS:eg 
Enclosure 
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Survey 



ID * 

Cost Analysis in Sdiool District 
Evaiuation Units 

This 1$ i survey to find wit (a) how school district evaluation units 
currently use cost data In their evaluations* and (b) what factors might 
Influence their use of fomal cost analysis methods* 

Please answer all of the questions. If you wish to coiifnent on any 
questions or to qualify your answers* please use the margins or a 
separate sheet of paper. 

Thank you for your assistance. 
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COST ANALYSIS IN SCHOOL DISTRICT EVALUATION UNITS 



Some school district evaluation units receive requests for cost 
information. In order to assist thew, we need a more complete 
understanding of what kinds of cost work are currently being conducted 
and what work is likely to be requested. 



1. Listed below are several purposes that cost analysis stud1|s can 
serve. Please write the number of cost studies conducted «or each 
purpose by your evaluation unit In the last five (5) years and the 
approximate number anticipated over the next five (5) years. 



NUMBER OF COST STUDIES 

ANTICIPATED 
DONE IN PAST OVER THE 

PURPOSE OF COST STUDY 5 YEARS NEXT 5 YEARS 

a. To describe the costs of a 
program or programs 

(cost description) _ 

b. To compare the costs of a 
program or programs with 
resources available to see 
if they are affordable 

(cost feasibility) 

c. To compare costs and 
outcomes for 2 or more 
programs » where outcomes 
are estimated 

(cost utility analysis) 

d. To compare the costs and 
outcomes for 2 or more 
programs » where outcomes 
are measured in dollars 

(cost benefit analysis) , 

e. To compare the costs and out-* 
comes for 2 or more programs, 
where outcomes are measured 
in test scores I behavioral 
ratings, etc. 

(cost effectiveness analysis) ^ 
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Is there currently a forwal expectation or requirement within the 
district that your unit do sooie fonn of cost analysis work? 



NO YES PLEASE EXPLAIN: 



Approxlieately what percent of your total 
evaluation unit resources are currently 
devoted to performing some form of cost 
analysis Mork? 

. PERCENT 

■ I J 



Do you anticipate that over the next 5 years there will be a formal 
expectation or requirement that your unit do some form of cost 
analysis work? 



NO YES PLEASE EXPLAIN: 



Approximately what percent of your total 
evaluation unit resources do you believe 
will be devoted to performing some form 
of cost analysis work over the next 5 
years? 

PERCENT 
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Some units report a number of Impediments that prevent Jhem f rom 
doing cost stSdIes. Please Indicate the degree to which the 
following factors restrain the number of cost studies done by yow 
evaluation unit. , 



UJ 

< 



Ul 
UJ 



TWESE FACTORS IMPEDE OUR OOIHG COST STUDIES: 

(Circle degree of agreement or disagreement i _ _ _ 
for each factor.) g g v> 

t;; •« o «/> s 

a. We are seldom asked to do cost studies . . SA A D SD OK 

b. Decision makers are not often Interested 

in actual cost Information 5A « u 

c. Decision makers do not often use the «. • n «;n mc 
results of cost studies SA A D SO ur. 

d. It takes too much time to conduct cost «« . n 5D DK 
studies • • • " 

e. It costs too much to conduct cost studies. SA A 0 SO DK 

f . Accurate cost data are usually not « a n 50 OK 
available ^■ 

q. Cost study results are Incomplete 

because it Is not possible to Include . n SO DK 

al 1 important cost factors * ° 

h. Accurate outcome data are usually not « » n SO DK 
available ^ " ^ 

i. It is difficult to relate cost data to . „ «5D DK 
educational outcomes SA A o 

J. Cost results do not tell managers how to . « rn qk 

improve program operations ^'^ " 

k. We lack staff with the technical capability ... ^ 50 
to conduct cost studies SA A 0 5D Vf. 

1. We lack available consultants or experts 5. . n 50 DK 
to help us conduct cost studies SA A 0 5D 

m. We do not have sufficient experience In . « 50 DK 

conducting cost studies SA A D SD DK 

n. We have few guidebooks, texts, or examples . - 5^ «,( 

to follow In conducting cost studies ... SA A 0 50 
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ru.. f.L^i^^"^ ANALYSIS STUDY HAS NOT SEEN CONDUCTED BY YOUR 
EVALUATION UNIT IN THE PAST FIVE (5) YEARS, SKIP TO QUESTION 6. 

^ . , - 



5. Now we would like to know about cost analysis studies that Rave been 
conducted by your unit . Think beck on all the cost studies that have 
been conducted in your unit during the past 5 years. Please provide 
• brief sunmary of the study you consider to be the "best" example of 
a cost analysis study conducted in your unit. 

«. The topic (e.g., bus leasing; a reading program) 



^! L$?** ^^^^^ description, cost feasibility, 

cost uttlity, cost benefit, cost effectiveness— see question 1 
for descriptions) 



c. The decision or actions resulting from the study 



d. Your assessment of the value of this particular study 



e. Major difficulties in implementing the study 



f . Title and year of the study report 



SKIP TO QUESTION 7 
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What would you say was the primary reason no cost analysis studies 
have been conducted by your unit in the past five (5) years? 



COST EFFECTIVENESS ANALYSIS 

One type of cost analysis procedure, COST EFFECTIVENESS ANALYSIS, 
compares all the costs (e.g., personnel, facilities, equipment) of 
two or more similar programs to measures of program outcomes (e.g., 
test scores, behavioral changes). 

CHECK YOUR HIGHEST LEVEL OF FAMILIARITY WITH COST EFFEQIVENESS 
ANALYSIS (check one): 

Have conducted studies using it 

Have studied or read about it 

Have minimal familiarity with it 

Have no knowledge of it 
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COST EFFEaiVENESS ANW.YS1S Is one of the m^it suitable ways of 
comparing the costs and outc(iiMs of one ed^tatlon program with 
another, but so far the technique has been used little In educational 
evaluation. To understand why tlUs technique Is not widely used In 
educational evaluation, we would /like your opinions about the 
following Items. ' 

DO YOU THINK THAT: / g 

(Circle degree of agreement or disagreement i S * 

for each factor.) / " ^ t w S I 

-J Ul _j S 

g Uj g l_ 

a. Cost effectiveness analysl^ /$ often S § p£ g 
unnecessary because declswn makers are «/> «£ o i/i S 
not Interested In relatind program costs 

and effects , / SA a d sd dk 

b. Cost effectiveness anamis Is often not 
feasible because comparative program 

studies are seldom po^ble sa A 0 SO OK 

c. Cost effectiveness analysis Is too 
specialized a technique to be generally 

applicable SA A 0 SO OK 

d. Cost effectiveness analysis Is so costly 
and complex that It Is warranted only for 

major studies SA A D SD OK 

e. Cost effytlveness analysis considers only 
a 11ml tC9 number of program outcomes and 

so does not represent true program effects. SA A 0 SO DK 

f . Cost effectiveness analysis Is of limited 
utility since It provides no Information ' 

on program procedures or local conditions. SA A 0 SO OK 

g. Cost effectiveness analysis Is difficult to 
do because of technical details (e.g., 
discount rates) and the need for sophisti- 
cated analysis procedures SA A 0 SD OK 

h. Cost-effectiveness analysis cannot be 
applied to educational programs without 

further "development of the method ..... SA A 0 SO OK 
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Finally. In order tp understand the differences among school district 
evaluation units wuh respect to cost analysis, we need to know a few 
facts about your urjlt. 

9. ^ow many students are enrolled In your district? 

/Number of students: 

10^ What was the total annual budget for your evaluation unit for the 
; 1983-1984 school year? 

Total budget: $ 

11. How many full-time equivalent (FTE) professional staff worked In your 
unit during the 1983-1984 school year? 

Total professional staff: " FTE 

12. Many evaluation units have responsibilities other than conductlnq 
evaluation studies. Please Indicate below the major responsibilities 
of your evaluation unit during 1983-1984. 

CHECK ALL MAJOR RESPONSIBILITIES: 

a. Evaluation studies 

b. Evaluation monitoring 

c. Testing programs 

d. Planning 

e. Research studies 

f. Consultation/technical assistance 

q. Policy analysis 

" " h. Needs assessment 
1. Information provision 

j. Other (please specify) 



Th^nk You . 

Please use the enclosed envelope to return this completed questionnaire 
to Or. Jana Kay Smith, Research on Evaluation Program, Northwest Regional 
Educational Laboratory, 300 S.W. Sixth Avenue, Portland* OR 97204. For 
Inquiries, call 1-800-547-6339. 
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